Automated Vocabulary Acquisition and Interpretation in Multimodal Conversational Systems
نویسندگان
چکیده
Motivated by psycholinguistic findings that eye gaze is tightly linked to human language production, we developed an unsupervised approach based on translation models to automatically learn the mappings between words and objects on a graphic display during human machine conversation. The experimental results indicate that user eye gaze can provide useful information to establish such mappings, which have important implications in automatically acquiring and interpreting user vocabularies for conversational systems.
منابع مشابه
Incorporating Temporal and Semantic Information with Eye Gaze for Automatic Word Acquisition in Multimodal Conversational Systems
One major bottleneck in conversational systems is their incapability in interpreting unexpected user language inputs such as out-ofvocabulary words. To overcome this problem, conversational systems must be able to learn new words automatically during human machine conversation. Motivated by psycholinguistic findings on eye gaze and human language processing, we are developing techniques to inco...
متن کاملCognitive Principles in Robust Multimodal Interpretation
Multimodal conversational interfaces provide a natural means for users to communicate with computer systems through multiple modalities such as speech and gesture. To build effective multimodal interfaces, automated interpretation of user multimodal inputs is important. Inspired by the previous investigation on cognitive status in multimodal human machine interaction, we have developed a greedy...
متن کاملSemantics-based Representation for Multimodal Interpretation in Conversational Systems
To support context-based multimodal interpretation in conversational systems, we have developed a semantics-based representation to capture salient information from user inputs and the overall conversation. In particular, we present three unique characteristics: fine-grained semantic models, flexible composition of feature structures, and consistent representation at multiple levels. This repre...
متن کاملModelling the Context of Learning Interactions in Intelligent Learning Environments
Contextual modals p. 15 Understanding context before using it p. 29 Epistemological contextualism : a semantic perspective p. 41 Task-realization models in contextual graphs p. 55 Context-dependent and epistemic uses of attention for perceptual-demonstrative identification p. 69 Utilizing visual attention for cross-modal coreference interpretation p. 83 Meaning in context p. 97 Descriptive nami...
متن کاملUser Language Behavior, Domain Knowledge, and Conversation Context in Automatic Word Acquisition for Situated Dialogue in a Virtual World
To tackle the vocabulary problem in conversational systems, previous work has applied unsupervised learning approaches on co-occurring speech and eye gaze during interaction to automatically acquire new words. Although these approaches have shown promise, several issues related to human language behavior and human-machine conversation have not been addressed. First, psycholinguistic studies hav...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2007